

Whose house is of glasse, must not throw stones at another - smokinn
http://gettys.wordpress.com/2010/12/06/whose-house-is-of-glasse-must-not-throw-stones-at-another/

======
jmillikin
This article is part of a series, which examine performance problems in high-
speed internet connections. Unfortunately, in addition to being split over
several pages, it is written in one of the most rambling and verbose styles
I've ever seen. The best single explanation is found not in the submitted
link, but in < [http://gettys.wordpress.com/2010/12/03/introducing-the-
crimi...](http://gettys.wordpress.com/2010/12/03/introducing-the-criminal-
mastermind-bufferbloat/) >

My background in networking isn't strong enough to fully understand the
problem, but here's an attempt:

\--

When using a fast Internet connection, in the 20+ Mb/s range, throughput and
latency for common residential use cases are both far worse than
expected[1,3,4]. The problem is masked from most users because Windows XP is
missing a feature required for maximum network performance, and because most
American residential Internet connections are slow. As more users upgrade
their operating systems and connection, significant performance will be lost.

The problem occurs due to excessive buffering, combined with the TCP window
scaling feature. When a network stack with support for TCP window scaling is
allowed to connect to a fast link, the link is saturated. Normally the dropped
packets should signal the stack to back off, but buffering in the stack and
local switch combine to cause massive latency (on the order of 10-20 seconds)
before packets begin dropping. This latency, which the author terms
"bufferbloat", is responsible for the performance degradation.

The author goes on to diagnose and correct excessive buffering in various
nodes; his described fixes are beyond my understanding, but indicate that
altering his Linux desktop's TCP queue parameters is sufficient to correct
many cases. Similar fixes to OS X and Windows can likely be applied by their
respective developers, once they become aware of the problem.

\--

The articles, in order:

[1] <http://gettys.wordpress.com/2010/10/02/first-puzzle-piece/>

[2] [http://gettys.wordpress.com/2010/10/13/browsers-and-tcp-
revi...](http://gettys.wordpress.com/2010/10/13/browsers-and-tcp-revisited/)

[3] [http://gettys.wordpress.com/2010/11/29/home-router-puzzle-
pi...](http://gettys.wordpress.com/2010/11/29/home-router-puzzle-piece-one-
fun-with-your-switch/)

[4] [http://gettys.wordpress.com/2010/12/02/home-router-puzzle-
pi...](http://gettys.wordpress.com/2010/12/02/home-router-puzzle-piece-two-
fun-with-wireless/)

[5] [http://gettys.wordpress.com/2010/12/03/introducing-the-
crimi...](http://gettys.wordpress.com/2010/12/03/introducing-the-criminal-
mastermind-bufferbloat/)

[6] [http://gettys.wordpress.com/2010/12/06/whose-house-is-of-
gla...](http://gettys.wordpress.com/2010/12/06/whose-house-is-of-glasse-must-
not-throw-stones-at-another/)

[7] [http://gettys.wordpress.com/2010/12/07/bufferbloat-and-
netwo...](http://gettys.wordpress.com/2010/12/07/bufferbloat-and-network-
neutrality-back-to-the-past/)

[8] [http://gettys.wordpress.com/2010/12/08/bufferbloat-
mitigatio...](http://gettys.wordpress.com/2010/12/08/bufferbloat-mitigations/)

[9] [http://gettys.wordpress.com/2010/12/09/bufferbloat-and-
conge...](http://gettys.wordpress.com/2010/12/09/bufferbloat-and-congestion-
collapse-back-to-the-future/)

[10] [http://gettys.wordpress.com/2010/12/13/mitigations-and-
solut...](http://gettys.wordpress.com/2010/12/13/mitigations-and-solutions-of-
bufferbloat-in-home-routers-and-operating-systems/)

~~~
StavrosK
Thank you for that. Can someone post a fix here, if any, for whatever OS they
know? I couldn't detect one in that monster of a post...

~~~
noonespecial
I've been using such a thing since about October of 2002 in linux. Its based
(well it was at first anyway) on a quick little hack called "wonder-shaper".

<http://lartc.org/wondershaper/>

I put it on the linux box that was routing my ISDN (don't laugh) and magic
happened.

I was then compelled to study the deepness that is LARTC and found much
enlightenment.

The path lies at <http://lartc.org>

~~~
StavrosK
Thanks for that, it's even included in Ubuntu. I'm just not sure if it will
work if I install it on all my computers individually, because I don't have a
router running Linux (pity me).

~~~
bobf
Indeed. It would be nice if dd-wrt or a variant supported a wonder-shaper sort
of fix..

------
JoachimSchipper
A quick read shows that this is hardly novel. If I missed something, please
point it out.

The issue is simple: broadband providers/modem manufacturers/... want to
advertise the largest download speeds they possibly can. To do this, they add
large buffers (such that they can always fill the pipe with bits).

However, they overdo this: by the time you've buffered a second worth of data,
new data can't get through the router in a timely fashion. This means that
VoIP sucks (one second latency), TCP sucks (it sends a limited amount of data
before waiting for ACKs - note the recent "Google floods clients with 12 TCP
packets at once" post on HN), and you may get random drops as well.

The solution is to rate-limit your outgoing internet connection to just a
little less than the actual rate, and do queue management on a competent
device (e.g. any unixish system should do). Prioritize VoIP, Quake, TCP ACKs,
etc, and deprioritize BitTorrent/FTP/anything with the appropriate QoS bits
set. Google for a guide relevant to your favourite implementation
(netfilter/pf/...)

~~~
jmillikin
I believe the author is discussing excessive buffering in the local hardware
and OS, not the ISP's. He specifically calls out Intel and some Linux drivers
as known causes. Additionally, he demonstrates that changing OS buffering
parameters corrects the problem in many cases (going from 1-2s latency to
30ms).

------
whatusername
The conclusion summs that article up much better than the title: "All
broadband technologies are suffering badly from bufferbloat, as are many other
parts of the Internet."

It's a little over my head -- any TCP/Networking experts want to chime in?

~~~
chaosmachine
_Normally, TCP waits for the buffer to exceed the maximum segment size before
sending any data. This creates serious delays when the two sides of the
connection are exchanging short messages and need to receive the response
before continuing. For example, the login sequence at the beginning of a
telnet session begins with the short message "Login", and the session cannot
make any progress until these five characters have been transmitted and the
response has been received. This process can be seriously delayed by TCP's
normal behavior when the message is provided to TCP in several send calls._

<http://en.wikipedia.org/wiki/Transmission_Control_Protocol>

Basically, there's a bunch of large TCP buffers between you and the rest of
the internet, which can fill up when you're trying to transfer large amounts
of data. When these buffers are full, important packets get queued up for long
periods of time, causing latency. This is why running something like
bittorrent causes your ping time to go up.

It's especially a problem when uploading, because upload speeds are often
capped at a tiny fraction of your download speed, so it takes much longer to
drain the buffers and get to the important packets.

See also this post:

[http://gettys.wordpress.com/2010/12/03/introducing-the-
crimi...](http://gettys.wordpress.com/2010/12/03/introducing-the-criminal-
mastermind-bufferbloat/)

------
kingkawn
A google search to better understand blufferbloat turned up exactly 0 entries.

~~~
Groxx
> _... blufferbloat ..._

Did you try removing the "l" from "buffer"?

~~~
JoachimSchipper
Come on, that kind of snark isn't helpful. And no, removing the l does not
seem to result in significantly more useful Google results.

~~~
Groxx
I was serious; "bufferbloat" finds 27,000 results, a far cry from zero. I
figured it was a spelling error.

Only upon seeing this comment did I check the results, which unfortunately do
seem to (nearly) all point back to these articles. I'm unsure if the remaining
ones (a fair amount can be found by adding "-gettys") refer to the same thing
or not, but there are a few other mentions, especially if you add a space.

~~~
rayval
I think it is because Jim Gettys coined the term very recently. It seems
likely that he is the very first person to discover this phenomenon.

Although there is a small chance that someone else also discovered it
independently but came up with their own phrase to describe it -- a word that
does not show up when you search for "bufferbloat".

The word "bufferbloat" is OK but it is not quite on target, imho. I don't have
a better suggestion, but I think a better alternative is out there somewhere.
Such a word would convey, not just the notion of bloatage, but also the notion
of bits getting stuck in the comfortable expanse of modern, obese, buffers,
fed at a rapid rate by modern hardware (and TCP stacks) on clean pipes.

Grasping at metaphors, I'm thinking of dampeners like bit potholes, quicksand,
flypaper, speed bumps, bit tarpits...

~~~
Groxx
"bit tarpit" has a nice ring to it. Or maybe "bithole". Though I'm now
inescapably, _desperately_ trying to come up with _something_ legitimate that
includes "tubes"...

edit: Tube Torpor!

