My favorite is not filling buffers before sending the packet and then everything with a crappy internet connection (dropped packets + high latency = TCP retransmissions) is suddenly doing 120kps because you're only sending 50 byte packets and every 10th one is getting lost. Meanwhile, that's a thread on your server not doing useful work.
Would you mind describing this problem more fully? I'm currently learning TCP more deeply and this sounds interesting but I don't quite understand the issue.
1. Wait until you have enough data to send in at least one packet (maximize throughput).
2. Send data as soon as you have it, even if it is less than a packet's worth. (minimize latency).
If you want to send a file, for example, you should use the first choice. A common or naive implementation might be to read the file in chunks of 9kb (about the size of a jumbo frame), send it to the socket, and then flush it. There are multiple problems with that approach.
1. Not everyone has jumbo frames, so if we had a common MTU of 1500 bytes on our link, that means you'd actually send ~6.25 packets worth of data.
2. Even if you were using jumbo frames, TCP headers use up that space. So, if you flush the entire frame's worth of data, you'd send something like 1.1 packets.
3. The filesystem might or might not give you the maximum bytes you request. If there is i/o pressure, you might only get back some random amount instead of the 9k you requested. So, if you flush the buffer to the TCP stack, you'd only send that random amount instead of what you assumed would be 9kb.
These mistakes generally send lots of small packets very quickly, which is fine when the link is fat and short. As soon as one packet gets lost, you're still sending a bunch of small packets, but now we have to stop and wait for the lost packet again before we can start handling more of them. So, if you are sending to someone on, say, airport/cafe wifi, they will have atrocious download speeds even though they have a pretty fat link (the amount of retransmissions due to wifi interference is a large % of the bandwidth). This is very similar to "head of line blocking" but on the link level.
I only had to learn about this fairly recently because my home is surrounded by a literal ton of wifi access points (over 20!) which causes quite a bit of interference. On wifi, I can get around 800mbps on the link but about every 10th packet needs to be retransmitted due to having so many access points around me (not to mention my region uses most of the 5G band for airport radar, so there are only a few channels available in that band). So, when applications/servers don't fill the buffers, I get about 50kpbs. When they do fill the buffers, I can get around 300mbps maximum, even though I have an 800mbps link.
