TL; DR version: packets get dropped when some buffers, at your local computer or a router between here and there get full.
I do not want to sound too critical -- the info is good for someone who never heard of UDP.
But I was hoping for more information. More substantiation on why full buffers are the main source of UDP drops (e.g., can smart throttling take some/most of the blame -- given the need to drop a packet, dropping a UDP is usually less painful than dropping TCP, etc.)? any quantitative numbers on sample network / hardware? etc.
Of course dropping traffic isn't the intended purpose of the protocol, but it's designed to allow unreliable communication when you're stuck with that situation.
Of course, I think that reference was in a book, so I can't find it now.
A few systems I saw would, in this case, heavily favor dropping UDP on the assumption that the applications using it can handle dropped packets in stride; plus, dropping TCP packets only causes more traffic in the short term due to retransmissions.
In this case the drops could have nothing to do with buffer sizes -- UDP packet can still be dropped even if it is the only UDP packet in a large buffer.
The infectious tone is part of it, but I think the bigger win is just how little Evans cares about what the reader/writer is "supposed" to know. Each post goes from a standing start with almost nothing presupposed all the way into the weedy details.
Regarding this post: it's getting close to a pretty big idea. Once you grok why packets get lost† --- congestion --- you're pretty close to understanding the big idea behind TCP, and how the Internet works and figures out how fast to send things, even though we're on crappy wi-fi connections connected to even crappier DSL lines connected to OC192 backbones.
† Fun additional reason we found when trying to build a next-gen "routed" IRC in the 90s: when routes change!
A few rabbit-holes to dive down:
The extra pain is that even though the huge datagram is lost, almost all of its data is transmitted. So you have a congested line and you are hammering it extra hard by resending the same data over and over.
Moral: unless you can guarantee low IP packet loss rates across the entire route, be very careful about large UDP packets.
Counter moral: I built a very nice VPN over TCP solution for a customer who insisted 10% packet drop on their network was fine. Completely fixed their large UDP packet based legacy system… which I had designed a decade before.
Modern OSes have a thing called black-hole detection, it is a good idea to have something similar in your protocol running on top of UDP.
If sending a packet syncronously, the system call would just hang until the buffers have drained enough.
How would you do it while receiving? Does the outside network just bang bits through the cable like a radio station? Then you'd have no choice but to save everything as fast as it comes in, lest you loose packets. Or is there, deep down on the lowest levels of the stack, actually some kind of request/response going on, even for UDP? For example at the level of Ethernet frames, or even individual bits? Like "here are some bytes. got them? - yes. - here are some more. got them? - (waiiiiting....) yes." Then you could just let the next router in the system wait while you drain your input buffer.
Even if there is no request/response going on, you could still view your incoming as the next routers outgoing. Configure the router to wait with sending until your client has drained the router's output buffer enough. (That would require the router to know how much data you can take.)
It sounds like you're trying to make UDP lossless. Use TCP instead.
What if I'm running VOIP? Or a real-time video game? I don't want to wait for the dropped datagram to be received before processing the next one. It's too late! It should have been received already. Skip it and use the next one.
> Does the outside network just bang bits through the cable like a radio station?
Basically, yes. Ethernet has no flow control. IP has no flow control. UDP has no flow control. TCP does.
> is there, deep down on the lowest levels of the stack, actually some kind of request/response going on, even for UDP?
No. Ethernet, as commonly deployed, has no need for this, but CSMA/CD is the retransmit part of the specification in case you are using coax tap or a hub. Each Ethernet frame also has a Frame Check Sequence, and the receiving node will discard the frame if it doesn't match up. It's not up to Ethernet to retransmit. Use TCP for that instead.
Except it does: https://en.wikipedia.org/wiki/Ethernet_flow_control
Whether people use it or not is another matter!
The messages can be up to around 64KB. I say "around" because if different OS's based on experiment did different things (this caused us much confusion). I think HPUX would drop them silently on send if they got to 62. at 64 it would return an error. Keep them below 60 to be safe.
Multicast means the router has to be set up correctly. When they muck with the network and add a hop, make sure your TTL (time to live) is set correctly on send. One of our OSs had a strange default to this.
I liked the all or nothing receive nature of UDP messages. Never a waiting and reading for the rest of the data like with TCP. You have no idea if the message got to where it was sent, sometimes you don't need that. Very low overhead too.
Multicast was the selling point for us. Send and everyone attached to the group gets the message. You can subscribe to the groups and see all the messages which makes debugging easier.
So UDP packets are as reliable as TCP packets in principle. TCP just hides the unreliability with flow control and retransmits.
The practical solution is to meter traffic (don't send faster than they can be processed on the receive end or faster than the tightest congestion bottleneck enroute). At the same time process as quickly as possible on the receive side - don't let a single buffer sit in the IP stack longer than absolutely necessary. E.g. receive on one higher priority thread and queue to a processing thread.
E.g. receive on one higher priority thread and queue to a processing thread.
Yes, the solution to buffer bloat is to only keep buffers at the ends, but whether that's in the kernel or in your application shouldn't matter in most of cases, if any at all. Better to just let the kernel handle it and not recreate the wheel.
Tweak the kernel send/receive buffers if you want. The defaults on Linux are usually too aggressive, but I don't see any need to do much more than that.
And as observed elsewhere in this thread, some OSs have tragically small buffers (128K). Its absolutely vital to keep those buffers from filling in ambitious apps.
I wrote audio/video/screenshare communications code for years. In the bursty situation I described, the whole point is to offload the ip stack buffer into the app buffer at high speed.
I've also written streaming media services for many years. RTP/RTCP, for example, supports adaptive rate limiting, though few implement it. The RTCP sender and receiver reports signal packet loss and jitter so that the sender can, e.g., dynamically decrease the bitrate. If implemented properly, buffering too many RTP packets can hurt the responsiveness of the dynamic adaptation, which can quickly lead to poorer quality. (Modern codecs help to mitigate this issue, but largely because the creators have spent a lot of time putting more adaptation features into the codecs and the low-level bitstream knowing that software higher up the stack is doing it wrong.)
For DNS, because Linux has a default 65KB (or greater!) buffer on UDP sockets, it's trivial to get huge packet loss when doing bulk asynchronous DNS queries. The application will quickly fill the deep UDP buffer; with the deep buffer the kernel will keep the ethernet NIC frame buffer packed, with the result that you'll see a ton of collisions on the ethernet segment and dropped UDP packets once the responses start rolling in. That results in a substantial fraction of the DNS queries have to retransmit, and because the retransmit intervals are so long, that means a bulk query operation that could have finished in a few seconds or less could take upwards of a minute as the stragglers slowly finish or timeout. Without the deep UDP pipelines, the ethernet segment would be less likely to hit capacity, would see fewer dropped packets, and so the aggregate time for the bulk query operation would be several times less.
Reducing the UDP output buffer is substantially easier than implementing heuristics or gating for ramping up the number of outstanding DNS queries. The latter, if well written, might be more performant, but just doing the former would alleviate most of the problem, allowing you to move on to more important tasks.
There are also software reasons. A network stack is configured for some maximum UDP size and a datagram exceeds that size. On Mac OS X, the default maximum is just a little over 9200 bytes; you have to
sudo sysctl -w net.inet.udp.maxdgram=65535
Datagrams can be dropped due to being unroutable. Some router gets rebooted, and so its peers drop packets. TCP connections keep going after a little pause; UDP sessions lose datagrams.
Lastly, IP datagrams are dropped if their TTL (time to live) decrements to zero as they cross a router, even if they are otherwise routable, not too big for any buffer, and with a correct checksum.
- Packet gets corrupted. A bit in the right place could cause it to be dropped or delivered to the wrong destination.
- Something goes wrong on the PHY layer, say a dodgy fiber connection between two nodes. It'll then get dropped due to Ethernet CRC mismatches.
- There are network nodes whose primary reason for existing is selectively dropping packets. For example traffic shapers, firewalls.
I imagine all the fancy routing protocols people use could end up dropping a packet. Maybe that can be lumped in with traffic shapers and firewalls?
Fun story about randomly dropping UDP: comcast did that when they first started doing traffic shaping and it ruined my team fortress classic games. I assume there must be a better way than dropping packets. Maybe buffers :) ?
There is a better way of doing shaping for TCP traffic, by making the sender reduce the rate of transmission by other means than dropping packets. For example via manipulating the receive window. (And other, more sophisticated variations on that). But these methods aren't applicable to UDP in general, at best to some small subset of UDP-based protocols.
Plenty of VoIP and video streaming protocols also handle RTT hikes.
Im surprised that nothing like this is mentioned, isnt that undergraduate university stuff?
"It's possible that you send a UDP packet in the internet, and it gets lost along the way for some reason. I am not an expert on what happens on the seas of the internet, and I am not going to go into this."
Indirectly; lots of collisions put stress on the buffer, so the probability that the buffer will be exhausted will increase.
Of course, if an intermediary router fails after sending the message, a UDP packet will also be silently dropped.
That being said, Wi-Fi retransmits on more than just collisions. Under the hood, Wi-Fi actually has it's own acknowledgement protocol beneath the Ethernet layer. The idea is that with electrical bus protocols like Thicknet, each tap has a pretty good view of the whole bus. For wireless you might see your own frames just fine, but the destination might not. So you need to retransmit not just when you see collisions, but when the destination hasn't explicitly acknowledged your packets.
This is extremely uncommon on 1Gbps NICs but is much more common on 10Gbps+ NICs. Also this type of dropping can happen on both RX and TX.
Also the author missed a set of buffers. The RX/TX rings on the NIC! These store packets on the NIC before they are moved to RAM (for RX) or sent on the wire (for TX). You can see them and configure their size using ethtool on Linux.
$ ethtool -g ens6
Ring parameters for ens6:
RX Mini: 0
RX Jumbo: 0
Current hardware settings:
RX Mini: 0
RX Jumbo: 0
Is this backwards, or does the faster NIC really drop more packets? That seems like it would be unfortunate.
I suppose if you point a 10Gbps UDP stream at 1Gbps NIC there will be drops but these drops will happen at the switch not at the interface which is a different type of dropping.
I know HTTP2(?) and/or SCTP moved in this direction, but seems no luck.
Windows, BSDs, OSX only buffers 1 packet per socket while it's ARPing
 Empirical testing.
 http://www.unix.com/man-page/FreeBSD/4/arp/ and similar for other BSDs
Of course packets could be lost so there actually was a paper based form you could fill in and re-request the missed data packet by fax (no-one ever did that, though).
I think by now they implemented a TCP/IP connection to request re-transmissions these days.
Actually, the causes of packet dropping are (AFAIK) exactly the same as in TCP. But in TCP, dropped packets are sent again.
One exception though: in UDP, delayed packets won't be waited for long before being dropped.
Unless you prioritize them differently.
Not so sure that urls should be any more than a uniqid - should they have semantic meaning? Discuss... No don't.