Hacker News new | comments | show | ask | jobs | submit login
A little bump in the wire that makes the Internet faster (apenwarr.ca)
156 points by eadmund 4 months ago | hide | past | web | favorite | 59 comments

The bufferbloat guy is much better at explaining this than I am. The problem needs a PR guy, and now it has one. I wrote this up in 1985 as RFC 970, "On packet switches with infinite storage", which is about why giant FIFO queues are bad and you need fair queuing at a bandwidth choke point. People who read RFCs and IEEE Trans. on Communications know about it. But I've been out of networking since 1986. (A non-networking startup wanted me to Do Something. I did. It worked out well. I never went back to networking.) So nobody was pushing this.

Fair queuing seems to have been an idea that was used for a while and then forgotten. It's been an obscure option on Cisco medium-speed routers for years, but only on by default for 2mb/s links and below.[1] Most of the attention on congestion control went into the TCP side, not the router side. There was too much attention to packet dropping, rather than packet reordering. In a router, you can do three things about congestion - drop, reorder, and delay. For decades, most of the action revolved around dropping, with Random Early Drop being popular. Reordering means you get to make decisions about what to reorder. That leads to policy issues, traffic shaping, quality of service, administrative interfaces, tuning knobs, and customer service/help desk calls. Which is why QoS never went much of anywhere outside of some tightly managed networks.

A big contribution of the bufferbloat guy is that he promotes the original dumb fair queuing - each source IP/source port/destination IP/destination port is one flow, and you service each flow equally. Nice and simple. No tuning. If that was in the middle of the network, you'd worry about people getting more bandwidth by gaming port numbers or something. But what's the point of gaming your own personal DSL router? In practice, the things that need really good latency on home DSL uplinks are acks, voice over IP, and game user action traffic, all of which are low-bandwidth. So the simple approach is good enough.

Early routers had barely enough CPU time to copy the bytes, let alone do fair queuing. So it wasn't in early routers. It's also a bit complex for a FPGA, so it's not in the big ones which route in hardware. (Nor is it that helpful there, because backbone routers don't buffer much.) That's how we got here.

It's embarrassing that this is still necessary to talk about. There's no excuse for big DSL operators deploying millions of routers with giant dumb FIFO buffers. Come on, AT&T.

A fair queuing add-on box is a definitive win only if the next link has a fixed data rate. DSL yes, cable maybe, cellular probably not. Cable modems at DOCSIS 3.1 and above are supposed to have DOCSIS-PIE [2], which is supposed to fix this.

Now will someone please do a startup that builds a little box? As a hobbyist thing this goes nowhere. As a little box you buy at WalMart, it does.

Or, AT&T guys, get your act together and ship this as standard. Thank you.

[1] https://www.cisco.com/c/en/us/td/docs/ios/12_2/qos/configura... [2] https://tools.ietf.org/html/draft-ietf-aqm-docsis-pie-02

John Nagle

I read through the first few pages thinking, "Get to the point: how did you detect the upload/download speed?" Finally,

> The only catch is... what upload/download speeds should we give to cake? Okay, I cheated for that one. I just asked my dad what speed his DSL link goes in real life, and plugged those in.

So the critical problem remains unsolved. If the ISP ever increases their speed cap, it goes to waste; if they decrease the speed cap, the fix stops working. I use a similar solution, but I have to turn it off and retest a few times a year, because Comcast sometimes changes the speed without notifying me.

One way to solve it is to look for TCP resends, over a diverse set of connections, and decrease the apparent line speed if there are "too many". The bloglog entry I posted earlier [0] goes over this.

[0] https://rkeene.org/projects/info/wiki/176

It's a device that goes between your router and your LAN that attempts to alleviate bufferbloat.

I think the blog really overdid it with the analogies.

I would have much preferred a network diagram with little buffers on the hosts filling up rather than the overwrought & unhelpful analogy.

Can I get an explanation why bufferbloat is? As in, why is it slowing things down? I got it's to do with big caches on devices? Or not? The article is extremely long without ever getting into it (I didn't click the article I was told not to click).

The TCP congestion control algorithm depends on dropped packets to calculate bandwidth. TCP will send more and more packets per second until they start dropping, then it will back off.

If there is a huge buffer, it will take a while for packets to start dropping, so the congestion control algorithm won't function optimally--it won't respond fast enough and lower the rate it sends packets until the buffer is full and packets start dropping. Once the buffer is full, packets have to wait in line in the buffer until they can be sent along so they take longer to get to their destination.

Eventually the buffer empties, but then the TCP congestion control algorithm starts increasing the rate it's sending packets and starts the problem over again. You end up with a very variable latency--depending on where in the cycle you are.

Actually, one TCP connection all by itself will do fine. It's multiple connections that will get you. In particular, doing an upload and a download at the same time will show terrible performance if there's a big FIFO buffer in the upload direction. The ACKs for the download, a small number of bytes, get stuck behind the bulk upload traffic. This runs the measured round trip time way up and causes the download TCP stream to slow.

This is why uploads interfere with downloads, even though they shouldn't.

>Actually, one TCP connection all by itself will do fine.

That depends on your metric. It isn't necessarily true if you're targeting stable latency for whatever reason.

Really good summary.

Slightly off topic....

He mentioned that over 20Mbps you really don’t notice better browsing speed. He’s right. I have gigabit internet - usually around 950Mbps up and down. Nothing takes advantage of it in the real world. The best real world throughout I’ve gotten is through my work VPN in the middle of the night or transferring something to AWS. Even then it’s only 100Mbps. BackBlaze tops out at around 70Mbps with multiple threads running.

Video streaming at its highest but rate is only 15Mbps.

I have had the opposite experience. I've had symmetric gigabit for a few years now and I monitor my upload and download rates in my bar on Linux. YouTube 4k videos can saturate around 500mbps (if you are scrubbing thru), backblaze can saturate all available bandwidth (but you need extremely high concurrency), game downloads can usually go to around 700mbps (on Steam, Origin and Epic launchers actually go faster). All in all I find it is useful day to day, but not quite as useful as you'd think. There are still way too many web servers that throttle download speed per connection or IP, however I've noticed there are less now than 4 years ago as gigabit consumer internet is becoming more common.

Worth noting this experience is from Phoenix, AZ (which is especially relevent because backblaze servers are in Sacramento and Phoenix).

I went from 100Mb Virgin Media Cable to 20Mb BT Fiber-to-the-Cabinet when I moved house (lack of other options in the new house). Normal internet browsing was unaffected, and I can still game while my partner watches Netflix.

The only time that it is tedious is when (for example) Overwatch ships a multi-GB update, and it hasn't auto-updated because my PC has been off. That used to be a "go downstairs and get a can of coke and it'll be pretty much done when you get back", but is now "go do something else for half an hour".

> Video streaming at its highest but rate

I expect you meant "bit rate", unless you're streaming a particular kind of video ;)

I was going to edit it...but your reply made it worth keeping the typo....

You ISP most likely has really crappy transit/peering. It doesn't help much if the last mile is gigabit, if the mile before that is also gigabit and shared with thousand(s) other users.

That’s true. The further out you choose servers for the speed tests, the worse the results. If I choose the server that is closest I get above 900. If I choose a server across the country, it’s about 150.

This is why most well-known speed test sites are useless, in that they are carefully gamed by the ISPs, so that they show you only the maximum speed you could reasonably achieve to their highly localized high speed network. It tells you little or nothing about what you could get to anywhere else.

The dslreports speed test is pretty good, and the one from fast.com should give you a pretty accurate picture of what you could see from the Netflix infrastructure.

Beyond that, you need to run your own CLI tools and be in control of both ends.

Just for a data point:

So I went to fast.com and it’s showing 150/73 using WiFi. Speedtest is showing 249/66. I am on the second floor. When I am right next to my router, speedtest maxes out at 371/354 wirelessly to the closest server.

Dslreports is 319/94.

This is on my iPhone 8 Plus. Usually my iPad is a little faster.

The benefit is more noticeable when you have more clients using the connection - big family with phones, tablets, desktops, etc.

Steam and torrents take advantage of all my gigabit bandwitdh. But generally if I'm using all of my bandwitdh, it's that I am consciously downloading/uploading stuff to do just that.

Are you sure there's not something on your network limiting your connection? When I had Gb many things were significantly faster. Most downloads would be at least 100Mbps. Steam would be 300-500Mbps.

Web browsing specifically has marginal improvements as the speed increases towards Gb. Latency has a larger impact as a percentage.

Speed tests show 950/950 wired, but I don’t do anything bandwidth intensive. My house is wired for gig e. My son may benefit with his PlayStation and Xbox.

The most bandwidth intensive thing I do is probably “download Linux ISO’s” on but torrent

What does slow it down is the 400 requests to go out and grab the JS, ads, graphics, etc.


Interesting topic, but the analogies actually make the article harder to understand.

Agreed. I found myself skipping over them in the hopes that the translations afterwards would be adequate. You’ve heard of bufferbloat, now try bloggerbloat!

Back in the day (2004 maybe?) I bought a device called the Broadband Booster (I think it was also called BBB). Two Ethernet ports (10/100 I think) and a power port. Really just a bump in your line.

Worked like a charm. I remember that it was specifically designed to get rid of buffer bloat, with traffic prioritization for streaming, VOIP, etc. I honestly think it was config-less though, so it doesn’t make sense unless it was also testing the speed on a regular basis and adjusting itself.

Edit: found it: Hawking Broadband Booster (HBB1) https://www.amazon.com/dp/B0009PLO0Q/ref=cm_sw_r_cp_api_jEiB...

Vendor Page:


Apparently based on Ubicom's StreamEngine technology, here's an old review:


This also points out the existence of the D-Link DI-102, which supposedly does the same thing.

I tried to replicate this setup with a Linksys E4200v1, but the ports/interfaces are labelled differently in the UI (vs. the author's D-link), so I'm having trouble getting this to work.

I got my Eth port 1 to bridge to WAN but it's not doing any shaping - my port 4 to Wan standard dual-nat shaping is working, however.

I find it amusing that anywhere with the option for wired internet is considered "rural". Is bufferbloat an issue for satellite based internet in the same way that it is for DSL?

It's a far bigger problem for satellite internet. The base RTT on satellite connections is far higher; think 500ms instead of 20ms. This means that any packet loss takes longer to recover from, which in turn means that there's a big incentive for the satellite provider to always buffer packets and never to drop them. Satellite internet also tends to be relatively slow compared to fixed line. It's a lot easier to trigger massive levels of bufferbloat on a 1Mbps connection than a 50Mbps one.

(I used to work on TCP acceleration boxes that among other things eliminated bufferbloat in satellite networks. But it's a far easier problem to solve when you're on the correct side of the bottleneck link.)

I have family in what is definitely rural Minnesota [1], and they have fiber to the home. I believe 50/50 is the base package, and when I looked into it, it was less than I pay for 25/25 in significantly less rural Iowa.

[1] The nearest town is https://en.wikipedia.org/wiki/Hillman,_Minnesota

Yup, rural co-ops can do a whole lot of things for their customers, when the big cable companies and telcos aren’t squeezing the sheeple for every red nanopenny.

Funny how that works out.

Is there any hope for setting something like this up for someone who doesn't really understand networks? I've got fiber service from my isp. Wires go in one end, and wifi comes out the other. I can get to the web-based config panel, but I wouldn't know how to go about making hardware changes.

Does anyone know what the cheapest hardware solution for buffer bloat is that can handle gigabit speeds?

I'm currently using an EdgeRouter Lite iii which reduces the download speed from gigabit to 130 Megabits per second when SQM is turned on.

You’re unlikely to experience bufferbloat at gigabit speeds, for the simple reason that your buffer will empty at gigabit speeds, which is about as fast as you can fill it.

Interesting. That matches my experience so far. It's not a symmetric connection so I have SQM running only on the uplink (40 megabits per second) at present.

Speaking of apenwarr, I ran the fast.com test and was sad to discover that my OnHub still has bufferbloat.

apenwarr has very little to do with OnHub. That said, OnHub is a wifi/firewall/router, and the bloat is in your modem, not in the router. OnHub can’t fix that unless it uses a configurable token bucket like in the article.

Has anyone tried this? I'm curious about experiences with this setup.

Uh, yeah, since about 2002.

apt-get install wondershaper

Specify your upload/download speed. All done.

It works really well if your upload and download speeds are predictable, as they were when I had cable. Now I have ADSL, and upload/download speeds vary with the weather (yes actually), and phase of the moon (probably), so it doesn't work so well, unless you set it to the lowest speed you are likely to see. (Don't do this on ADSL - it isn't pretty.)

Note that CAKE is better than wondershaper and just as easy to use, but wondershaper is certainly better than nothing.

It should be noted that wondershaper is now 16 years old, and it should be no surprise at all that there might be something a little better out there. It is fairly comprehensibly rubbished in https://www.bufferbloat.net/projects/bloat/wiki/Wondershaper...

Interesting topic, I guess.

However this is too much editorializing - I don't bother reading any more with this style of writing.

>Okay, new analogy: oil pipelines! Wait, those are unfashionable now too. Uh... beer taps... no, apparently beer is bad for diversity or something... chocolate fountains!

This writing style is really unnecessary.

You installed rate limiter that drops packets and doesn't buffer.

You don't need to write it like you're writing ad copy "This one little secret".

I actually find this style of writing interesting and engaging. It makes a mundane topic like rate limiters, easy to understand (along with the topics related to it). Not everyone enjoys the same styles of writing, but if I’m learning something new, I want it to be engaging.

> if I’m learning something new

If I'm reading something to learn something new, I want to end up learning about it by the end of it or I hit the limit of time I have to spend on learning about it.

A couple more details:

- The device is a D-Link DIR-825 running openwrt 18.06.0 with a pair of ports bridged so it can be dropped into place w/o having to reconfigure anything on the network.

- Cake[0] is being used for rate limiting.

- If you only care about the details of the setup without all the prefatory text skip to https://apenwarr.ca/log/?m=201808#openwrt

[0] https://www.bufferbloat.net/projects/codel/wiki/Cake/

Haha yea why'd that guy on vacation write with any sort of style... Everything should read like an RFC and look like notepad and we are robots that only consume information.

It's unnecessary if you're trying to write technical reference docs. That was not the goal of the author.

This style was more enjoyable to read and gave more insight into the author and their perspective. I appreciated it and read the whole thing.

Blogs are such a waste. I dont need to hear your life story and opinions.

Post the situation, the plan, and details/documentation.

I do not have the time to read a literal 6,000 words.

This comment breaks the site guidelines, which ask: "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."


> A good critical comment teaches us something.

How about that the format for this post was bad? Dont use blog formats when teaching a lesson.

Very quickly you will lose feedback if you downvote honest advice.

>I do not have the time to read a literal 6,000 words.

Don't read it then. It wasn't written for you or anyone else specifically.

Agreed, this seems like another 'writing to write' kinda thing. Like a school assignment.

Not functional.

The situation: The author improved internet functionality using what he knew about networking.

The plan: Write about what they know and how they were able to improve their parent's rural internet experience using layman's terms and analogies so that more people that suffer this situation might benefit from the improvement.

Details/Documentation: Read the submission.

damn, this is what this guy does on vacation? shit! I spend most of my time exploring the area (hiking, beaching etc) and usually have a couple brews in me by noon :p

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact