
A little bump in the wire that makes the Internet faster - eadmund
https://apenwarr.ca/log/?m=201808
======
Animats
The bufferbloat guy is much better at explaining this than I am. The problem
needs a PR guy, and now it has one. I wrote this up in 1985 as RFC 970, "On
packet switches with infinite storage", which is about why giant FIFO queues
are bad and you need fair queuing at a bandwidth choke point. People who read
RFCs and IEEE Trans. on Communications know about it. But I've been out of
networking since 1986. (A non-networking startup wanted me to Do Something. I
did. It worked out well. I never went back to networking.) So nobody was
pushing this.

Fair queuing seems to have been an idea that was used for a while and then
forgotten. It's been an obscure option on Cisco medium-speed routers for
years, but only on by default for 2mb/s links and below.[1] Most of the
attention on congestion control went into the TCP side, not the router side.
There was too much attention to packet dropping, rather than packet
reordering. In a router, you can do three things about congestion - drop,
reorder, and delay. For decades, most of the action revolved around dropping,
with Random Early Drop being popular. Reordering means you get to make
decisions about what to reorder. That leads to policy issues, traffic shaping,
quality of service, administrative interfaces, tuning knobs, and customer
service/help desk calls. Which is why QoS never went much of anywhere outside
of some tightly managed networks.

A big contribution of the bufferbloat guy is that he promotes the original
dumb fair queuing - each source IP/source port/destination IP/destination port
is one flow, and you service each flow equally. Nice and simple. No tuning. If
that was in the middle of the network, you'd worry about people getting more
bandwidth by gaming port numbers or something. But what's the point of gaming
your own personal DSL router? In practice, the things that need really good
latency on home DSL uplinks are acks, voice over IP, and game user action
traffic, all of which are low-bandwidth. So the simple approach is good
enough.

Early routers had barely enough CPU time to copy the bytes, let alone do fair
queuing. So it wasn't in early routers. It's also a bit complex for a FPGA, so
it's not in the big ones which route in hardware. (Nor is it that helpful
there, because backbone routers don't buffer much.) That's how we got here.

It's embarrassing that this is still necessary to talk about. There's no
excuse for big DSL operators deploying millions of routers with giant dumb
FIFO buffers. Come on, AT&T.

A fair queuing add-on box is a definitive win only if the next link has a
fixed data rate. DSL yes, cable maybe, cellular probably not. Cable modems at
DOCSIS 3.1 and above are supposed to have DOCSIS-PIE [2], which is supposed to
fix this.

Now will someone please do a startup that builds a little box? As a hobbyist
thing this goes nowhere. As a little box you buy at WalMart, it does.

Or, AT&T guys, get your act together and ship this as standard. Thank you.

[1]
[https://www.cisco.com/c/en/us/td/docs/ios/12_2/qos/configura...](https://www.cisco.com/c/en/us/td/docs/ios/12_2/qos/configuration/guide/fqos_c/qcfwfq.html)
[2] [https://tools.ietf.org/html/draft-ietf-aqm-docsis-
pie-02](https://tools.ietf.org/html/draft-ietf-aqm-docsis-pie-02)

John Nagle

------
p1mrx
I read through the first few pages thinking, "Get to the point: how did you
detect the upload/download speed?" Finally,

> The only catch is... what upload/download speeds should we give to cake?
> Okay, I cheated for that one. I just asked my dad what speed his DSL link
> goes in real life, and plugged those in.

So the critical problem remains unsolved. If the ISP ever increases their
speed cap, it goes to waste; if they decrease the speed cap, the fix stops
working. I use a similar solution, but I have to turn it off and retest a few
times a year, because Comcast sometimes changes the speed without notifying
me.

~~~
rkeene2
One way to solve it is to look for TCP resends, over a diverse set of
connections, and decrease the apparent line speed if there are "too many". The
bloglog entry I posted earlier [0] goes over this.

[0]
[https://rkeene.org/projects/info/wiki/176](https://rkeene.org/projects/info/wiki/176)

------
learc83
It's a device that goes between your router and your LAN that attempts to
alleviate bufferbloat.

I think the blog really overdid it with the analogies.

~~~
rplnt
Can I get an explanation why bufferbloat is? As in, why is it slowing things
down? I got it's to do with big caches on devices? Or not? The article is
extremely long without ever getting into it (I didn't click the article I was
told not to click).

~~~
learc83
The TCP congestion control algorithm depends on dropped packets to calculate
bandwidth. TCP will send more and more packets per second until they start
dropping, then it will back off.

If there is a huge buffer, it will take a while for packets to start dropping,
so the congestion control algorithm won't function optimally--it won't respond
fast enough and lower the rate it sends packets until the buffer is full and
packets start dropping. Once the buffer is full, packets have to wait in line
in the buffer until they can be sent along so they take longer to get to their
destination.

Eventually the buffer empties, but then the TCP congestion control algorithm
starts increasing the rate it's sending packets and starts the problem over
again. You end up with a very variable latency--depending on where in the
cycle you are.

~~~
Animats
Actually, one TCP connection all by itself will do fine. It's multiple
connections that will get you. In particular, doing an upload and a download
at the same time will show terrible performance if there's a big FIFO buffer
in the upload direction. The ACKs for the download, a small number of bytes,
get stuck behind the bulk upload traffic. This runs the measured round trip
time way up and causes the download TCP stream to slow.

This is why uploads interfere with downloads, even though they shouldn't.

~~~
learc83
>Actually, one TCP connection all by itself will do fine.

That depends on your metric. It isn't necessarily true if you're targeting
stable latency for whatever reason.

------
scarface74
Slightly off topic....

He mentioned that over 20Mbps you really don’t notice better browsing speed.
He’s right. I have gigabit internet - usually around 950Mbps up and down.
Nothing takes advantage of it in the real world. The best real world
throughout I’ve gotten is through my work VPN in the middle of the night or
transferring something to AWS. Even then it’s only 100Mbps. BackBlaze tops out
at around 70Mbps with multiple threads running.

Video streaming at its highest but rate is only 15Mbps.

~~~
nfriedly
> Video streaming at its highest but rate

I expect you meant "bit rate", unless you're streaming a particular kind of
video ;)

~~~
scarface74
I was going to edit it...but your reply made it worth keeping the typo....

------
tqkxzugoaupvwqr
Interesting topic, but the analogies actually make the article harder to
understand.

~~~
justwalt
Agreed. I found myself skipping over them in the hopes that the translations
afterwards would be adequate. You’ve heard of bufferbloat, now try
bloggerbloat!

------
jtbayly
Back in the day (2004 maybe?) I bought a device called the Broadband Booster
(I think it was also called BBB). Two Ethernet ports (10/100 I think) and a
power port. Really just a bump in your line.

Worked like a charm. I remember that it was specifically designed to get rid
of buffer bloat, with traffic prioritization for streaming, VOIP, etc. I
honestly think it was config-less though, so it doesn’t make sense unless it
was also testing the speed on a regular basis and adjusting itself.

Edit: found it: Hawking Broadband Booster (HBB1)
[https://www.amazon.com/dp/B0009PLO0Q/ref=cm_sw_r_cp_api_jEiB...](https://www.amazon.com/dp/B0009PLO0Q/ref=cm_sw_r_cp_api_jEiBBb9R5CFVG)

~~~
no_identd
Vendor Page:

[https://hawkingtech.com/product/hbb1/](https://hawkingtech.com/product/hbb1/)

Apparently based on Ubicom's StreamEngine technology, here's an old review:

[https://web.archive.org/web/20050829204222/http://www.tomsne...](https://web.archive.org/web/20050829204222/http://www.tomsnetworking.com/Reviews-210-ProdID-
HBB1-1.php)

This also points out the existence of the D-Link DI-102, which supposedly does
the same thing.

------
cloneman
I tried to replicate this setup with a Linksys E4200v1, but the
ports/interfaces are labelled differently in the UI (vs. the author's D-link),
so I'm having trouble getting this to work.

I got my Eth port 1 to bridge to WAN but it's not doing any shaping - my port
4 to Wan standard dual-nat shaping is working, however.

------
0xffff2
I find it amusing that anywhere with the option for wired internet is
considered "rural". Is bufferbloat an issue for satellite based internet in
the same way that it is for DSL?

~~~
war1025
I have family in what is definitely rural Minnesota [1], and they have fiber
to the home. I believe 50/50 is the base package, and when I looked into it,
it was less than I pay for 25/25 in significantly less rural Iowa.

[1] The nearest town is
[https://en.wikipedia.org/wiki/Hillman,_Minnesota](https://en.wikipedia.org/wiki/Hillman,_Minnesota)

~~~
bradknowles
Yup, rural co-ops can do a whole lot of things for their customers, when the
big cable companies and telcos aren’t squeezing the sheeple for every red
nanopenny.

Funny how that works out.

------
recursive
Is there any hope for setting something like this up for someone who doesn't
really understand networks? I've got fiber service from my isp. Wires go in
one end, and wifi comes out the other. I can get to the web-based config
panel, but I wouldn't know how to go about making hardware changes.

------
resters
Does anyone know what the cheapest hardware solution for buffer bloat is that
can handle gigabit speeds?

I'm currently using an EdgeRouter Lite iii which reduces the download speed
from gigabit to 130 Megabits per second when SQM is turned on.

~~~
apenwarr
You’re unlikely to experience bufferbloat at gigabit speeds, for the simple
reason that your buffer will empty at gigabit speeds, which is about as fast
as you can fill it.

~~~
resters
Interesting. That matches my experience so far. It's not a symmetric
connection so I have SQM running only on the uplink (40 megabits per second)
at present.

------
rkeene2
I have a similar rant:
[https://rkeene.org/projects/info/wiki/176](https://rkeene.org/projects/info/wiki/176)

------
wmf
Speaking of apenwarr, I ran the fast.com test and was sad to discover that my
OnHub still has bufferbloat.

~~~
apenwarr
apenwarr has very little to do with OnHub. That said, OnHub is a
wifi/firewall/router, and the bloat is in your modem, not in the router. OnHub
can’t fix that unless it uses a configurable token bucket like in the article.

------
larrydag
Has anyone tried this? I'm curious about experiences with this setup.

~~~
mnw21cam
Uh, yeah, since about 2002.

apt-get install wondershaper

Specify your upload/download speed. All done.

It works really well if your upload and download speeds are predictable, as
they were when I had cable. Now I have ADSL, and upload/download speeds vary
with the weather (yes actually), and phase of the moon (probably), so it
doesn't work so well, unless you set it to the lowest speed you are likely to
see. (Don't do this on ADSL - it isn't pretty.)

~~~
wmf
Note that CAKE is better than wondershaper and just as easy to use, but
wondershaper is certainly better than nothing.

~~~
mnw21cam
It should be noted that wondershaper is now 16 years old, and it should be no
surprise at all that there might be something a little better out there. It is
fairly comprehensibly rubbished in
[https://www.bufferbloat.net/projects/bloat/wiki/Wondershaper...](https://www.bufferbloat.net/projects/bloat/wiki/Wondershaper_Must_Die/)

------
unixhero
Interesting topic, I guess.

However this is too much editorializing - I don't bother reading any more with
this style of writing.

>Okay, new analogy: oil pipelines! Wait, those are unfashionable now too.
Uh... beer taps... no, apparently beer is bad for diversity or something...
chocolate fountains!

------
ars
This writing style is really unnecessary.

You installed rate limiter that drops packets and doesn't buffer.

You don't need to write it like you're writing ad copy "This one little
secret".

~~~
samschooler
I actually find this style of writing interesting and engaging. It makes a
mundane topic like rate limiters, easy to understand (along with the topics
related to it). Not everyone enjoys the same styles of writing, but if I’m
learning something new, I want it to be engaging.

~~~
falsedan
> _if I’m learning something new_

If I'm reading something to learn something new, I want to end up learning
about it by the end of it or I hit the limit of time I have to spend on
learning about it.

------
mkirklions
Blogs are such a waste. I dont need to hear your life story and opinions.

Post the situation, the plan, and details/documentation.

I do not have the time to read a literal 6,000 words.

~~~
dang
This comment breaks the site guidelines, which ask: " _Please don 't post
shallow dismissals, especially of other people's work. A good critical comment
teaches us something._"

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

~~~
mkirklions
> A good critical comment teaches us something.

How about that the format for this post was bad? Dont use blog formats when
teaching a lesson.

Very quickly you will lose feedback if you downvote honest advice.

------
brootstrap
damn, this is what this guy does on vacation? shit! I spend most of my time
exploring the area (hiking, beaching etc) and usually have a couple brews in
me by noon :p

