
TCP and BBR [pdf] - pjf
https://ripe76.ripe.net/presentations/10-2018-05-15-bbr.pdf
======
ejrv
I've been running BBR on both my web server and my own laptop for quite some
time now with good results. It's included in the mainline Linux kernel - if
you have kernel 4.9 or newer you can probably trial BBR yourself with these
sysctls:

    
    
      net.ipv4.tcp_congestion_control=bbr
      net.core.default_qdisc=fq
    

Most vaguely recent distributions (Debian stable, Ubuntu, etc) now support it,
and a few (like Solus) even default to using BBR.

~~~
nieksand
I've been interested in BBR since the ACM Queue article and finally turned it
on yesterday on my home workstation.

But slide 27 in the slide deck is incredibly troubling, where it shows BBR
completely destroying the throughput of a neighboring host using Cubic. (Cubic
being the default on modern Linux).

Even if everyone upgrades to BBR, slide 30 shows more trouble in paradise with
BBR not getting to a stable bandwidth allocation. I have a hard time
reconciling this with the ACM Queue article, since supposedly Google saw great
results rolling it out at Youtube.

ACM Queue article:
[https://queue.acm.org/detail.cfm?id=3022184](https://queue.acm.org/detail.cfm?id=3022184)

~~~
Arnt
Youtube doesn't have neighbouring senders using Cubic, and doesn't try to fill
the path's narrowest hop. It tries to send the video at exact playback speed
and if possible at high quality, and its notions of success will reflect that.

IIRC (I read the paper too) the Youtube numbers assumed that the bandwidth
available and needed are both largely stable, and that their success was in
using that better.

Ie. they assumed that if 4mbps appeared available most of the time, then it
really was available all of the time. If the capacity of the narrowest point
is 10Mbps and 4mbps youtube streaming briefly competes against a bulk download
from a Cubic sender, then BPR would hold the Cubic-using download to 60%,
which Youtube might count as success and you might or might not.

------
remcob
Delay based congestion control is the basis of LEDBAT (RFC6817) as used by
BitTorrent over UDP. The claimed advantage of LEDBAT is that it is submissive
to TCP, instead of dominating like BBR. This allows BitTorrent to run in the
background without hurting your videocalls.

What is different about BBR that makes it dominate while LEDBAT subsides?

------
Jedd
It's interesting to compare the work being done on BBR in (relatively) recent
times with the kinds of results coming out of various implementation of
SCPS[1]. In general, the biggest pain with conventional algorithms is their
boom/bust cycle and (re)ramp-up speeds. Satellite characteristics provide some
fascinating playgrounds.

[1]
[https://en.wikipedia.org/wiki/Space_Communications_Protocol_...](https://en.wikipedia.org/wiki/Space_Communications_Protocol_Specifications)

------
bogomipz
The article states the following about TCP Reno:

"It over-corrects on loss and leaves available path capacity idle

– 10Gbps rates over 100ms RTT demands a packet loss rate of less than
0.000003%

– An average 1% loss rate over a 100ms RTT can’t operate faster than 3Mbps"

Can someone explain the math there to me? I looked at the slides and I'm not
seeing it. Am I missing something really obvious?

~~~
jsnell
An additive increase/multiplicative decrease congestion control algorithm has
the property that the effect of a packet loss gets larger as the congestion
window increases. There will be a point where the upward and downward forces
on the congestion window balance out. The actual congestion window will
oscillate around that.

Assuming packet loss halves the congestion window while a packet delivery
increases the congestion window by 1/cwnd, the break-even point for 1% packet
loss is at a congestion window of 14. (1% change of reducing congestion window
by 7, 99% chance of increasing it by 0.07).

At 100ms RTT the average delivery rate will then be about 14 packets/rtt *
1500 bytes/packet * 10 rtts/second = 200kB/s.

What if we cut down the packet loss rate by a factor of 10, to 0.1%? It won't
give a 10x speed increase; instead the break-even point for the congestion
window will be about 45; only a 3x increase in speed. If RTT is constant,
increasing speed by 10x would require reducing packet loss by 100x.

(The math here is simplified, but should suffice for illustrating this).

~~~
bogomipz
Thank you yes this makes perfect sense, this is indeed the sawtooth pattern
you see with AIMD. Cheers.

------
bullen
ACK frequencies should be configurable, solving both bottlenecks and making
UDP less relevant at the same time:

[https://www.ietf.org/id/draft-add-ackfreq-to-
tcp-00.txt](https://www.ietf.org/id/draft-add-ackfreq-to-tcp-00.txt)

~~~
stiglitz
That configuration is already possible in Windows with the
“DelayedAckFrequency” parameter and in Linux with the “quickACK” setting, but
it’s unrelated to congestion control, and it will neither “solve” bottlenecks
nor “turn TCP into UDP.”

Congestion control (the purpose of BBR) deals with the bottleneck problem.
Delayed ACKs are an optimization to reduce protocol overhead. Indeed there are
problems with delayed ACKs when combined with Nagle’s algorithm and an
application that doesn’t send a steady stream of data, but it has nothing to
do with the topic of this thread.

~~~
bullen
I need to set ack frequency to never or inifiny (say 1000+) packets per ack.
so these are not usable for me.

TCP_QUICKACK seems to be a boolean (basically 1 or 2 packets per ack)

DelayedAckFrequency goes from 10 to 600 but it's millisecs!?

You don't think acks. have an impact on congestion?

