
The Linux kernel MultiPath TCP project - TallGuyShort
http://multipath-tcp.org/
======
songgao
Multi-path TCP is not only about throughput, but also about providing
redundancy to improve stability. And to me, redundant stream is more
attractive.

It happens a lot when I am with my iPhone at boundary of a Wi-Fi network
coverage, the network is very unstable with oscillations between Wi-Fi and 4G.
The phone stays in the status with connected Wi-Fi but it's actually not
working. If there was multi-path TCP, mobile devices can establish
simultaneous streams on both Wi-Fi and 4G. When Wi-Fi is not responsive, the
data stream can be automatically offloaded onto 4G seamlessly without causing
latency in application layer.

Another protocol that may achieve this is SCTP[1]. And it's been around for
years[2]. But migrating existing services to another transport protocol is a
challenge, which makes it difficult to adopt SCTP. I'm curious how multi-path
handles this. Is it an extension to TCP that can handle backward compatibility
with TCP, or yet another new transport protocol?

[1]:
[http://en.wikipedia.org/wiki/Stream_Control_Transmission_Pro...](http://en.wikipedia.org/wiki/Stream_Control_Transmission_Protocol)

[2]: <http://www.sctp.org/implementations.html>

\---

EDIT: More details about its design decisions and goals: <http://multipath-
tcp.org/data/MultipathTCP-netsys.pdf>

~~~
signa11
with two antennas (antennae ?) simultaneously broadcasting/receiving, i think
battery life would be pretty low...

~~~
Dylan16807
In my experience a data link takes far more power than wifi, so the extra use
won't be very notable.

~~~
krzyk
I think that the opposite is true (but depends on the type of connection).
Mobile stations are focused on low power, they were designed that way from the
beginning (e.g. they constantly try to reduce the transmit power if you have
"too good" reception so it will be just good and use a lot less power) unlike
wifi.

I remember the first mobile prototypes that were used around Motorola were
dying of battery after few hours if used wifi instead of mobile data (that was
about 8 years ago - so probably we are talking GPRS here).

------
mmastrac
Since the site is having trouble:

MultiPath TCP (MPTCP) is an effort towards enabling the simultaneous use of
several IP-addresses/interfaces by a modification of TCP that presents a
regular TCP interface to applications, while in fact spreading data across
several subflows. Benefits of this include better resource utilization, better
throughput and smoother reaction to failures. Slides - explaining MultiPath
TCP - are available in .pdf and .pptx format. You can also have a look at our
Google Techtalk about MPTCP. The IP Networking Lab is implementing MPTCP in
the Linux Kernel and hosting it on this website for users, testers and
developers. For questions, feedback,... please contact us at the mptcp-dev
Mailing-List

<http://multipath-tcp.org/data/MultipathTCP-netsys.pdf>

[http://www.youtube.com/watch?feature=player_embedded&v=V...](http://www.youtube.com/watch?feature=player_embedded&v=VWN0ctPi5cw)

~~~
diroussel
Maybe the site should be using MultiPath TCP?

~~~
cpaasch
It is using MPTCP. We are just having right now an extremly high visitor-
count. The server is only running on a tiny XEN-VM and thus it becomes
overloaded.

------
trotsky
Seems like this is extremely wireless carrier friendly. Once you're able to do
transparent tcp handoffs your ability to exploit unlicensed spectrum to
offload traffic from your saturated licensed bands without a very obvious
service interruption gets much easier. If you deploy devices set to auto
associate with a whitelisted set of provider and partner hotspots as soon as
the connection is up you can offload the bulk of it, letting you serve more
customers with the same amount of spectrum. Addressing heavy demand locations
with wifi is way cheaper than minicells or DAS systems and wouldn't involve
any regulatory approval or backhaul requirements.

I had to laugh when I saw the charts of LTE+wifi beating wifi alone in
throughput. Even if the phone os was setup to try to exploit that, any
wireless provider able to tie their own shoelaces is going to monitor
multipath traffic like that and once they infer your alternate link is
reasonably fast they'll start intentionally congesting their side of the path
to force the bulk onto the local route.

I wonder what sort of additional profiling this could enable. Right now my
phone is on 3g and my dual homed wifi network. It seems possible that being
able to see my traffic from wireless, wireline ip4 and via my ip6 tunnel
provider at the same time along with each path's latency would leak more data
than existing hard handoffs.

~~~
yxhuvud
It will probably be usable from the perspective of a mobile carrier, but
really, the hard part is the 'transparent tcp handoff' part. They already know
all too well how nice it is to offload traffic to wifi. Two problems exist,
and that most carriers don't have the servers or the wifi nets to support it
nicely, and that mobile phones doesn't support automatic and transparent
handover in a nice way (iOS 5 brought support though, so it is not all bad.
Android support is mostly lacking though).

Also, wifi already tend to be so much better that the bulk of the traffic
would already go that way if possible without carrier interference.

~~~
trotsky
This is the main feature of the design, transparent handoff smoother than
anything today, without any existing tcp connections dropping and faster
throughput recovery. They demo skype handing off from 3g to wifi keeping the
call alive and only suffering ~2seconds of delay during transition.

The project has a number of working kernels for various android phones
available.

Carriers would widely deploy ISM nets or whitespace if the handoff was
painless and your streaming video worst case lagged out a couple of seconds.

~~~
yxhuvud
There are levels of transparency. The level you are talking about still
require the wifi to either be entiredly open or to have the authenticaten
details stored in the phone manually. The level I'm talking about is where
your phone automatically authenticates using the sim card when you happen to
go near an access point, and then manage to do a handoff _with no manual steps
involved_.

I'm not saying multipath TCP will be very nice for mobiles, but it will mostly
be home and work networks that will be used, just like how it is for wifi use
in the mobiles today.

As for 2 second handoff being smoother than anything else out there today,
that is just not true. The situation for phones doing handoff with EAP-SIM/AKA
is pretty similar if not better.

~~~
trotsky
For what it's worth that's two seconds between when one signal was
unexpectedly cut, detected, shifted to the other, windows scaled up for a
mulimedia stream and the app buffered enough to decide to start playing audio
again.

Something considered a two second handoff is going to result in a far longer
stream interruption than that in any method I've seen used especially
considering the app is not assisting with the process or getting state
transition info which it could be if built with mptctp aware libs.

------
zokier
I wonder how difficult it would be to set up a MPTCP gateway/VPN, so that you
could get the benefits of MPTCP all the time without requiring MPTCP support
from the server you are connecting to.

eg:

    
    
        +--------+              +---------+          +---------+
        | Smart- +----WiFi----->| Gateway +---1Gb--->| Youtube |
        | phone  +-----3G------>|         |          |         |
        +--------+              +---------+          +---------+
                      MPTCP                   TCP

~~~
trotsky
It looks like you can do that with a proxy approach, there are active ietf
drafts concerning proxies:

[http://tools.ietf.org/html/draft-hampel-mptcp-proxies-
anchor...](http://tools.ietf.org/html/draft-hampel-mptcp-proxies-anchors-00)

Encapsulating tcp/udp inside it probably doesn't work very due to conflicts
between the layers of buffers and congestion control.

------
profquail
There's also a Multipath TCP patch for FreeBSD 10-CURRENT:

<http://caia.swin.edu.au/urp/newtcp/mptcp/tools.html>

Slides with more info:

<https://www.ietf.org/proceedings/84/slides/slides-84-mptcp-2>

------
corbet
LWN's overview: <http://lwn.net/Articles/544399/>

~~~
aray
This is a great short summary, followed by the Tech Talk (on youtube) which
goes into a bit more depth about the how and why of the design considerations.

What isn't mentioned is basically, because the change is 10KLoC to the kernel
(unprecedentedly huge change for just the functionality they added) it hasn't
even been submitted yet, and mainline hasn't even considered it yet.

I'd love to see it hit mainline, or parts of it start getting submitted, but
the biggest thing will probably be re-writing the whole thing to have a much
smaller impact on the kernel, while still providing the functionality they
need (e.g. use the TCP backoff instead of their own, etc).

------
jsaxton86
What advantages does this have over existing link aggregation implementations?

Edit: one advantage I can think of is you don't have to ask your users to set
up a bond0 interface, for example. Any other advantages?

~~~
trotsky
Current layer 2 link aggregation is effective for lots of connections with
different source/destination pairs because the switches have limited resources
and require simple schemes. It works fine for public servers and transit
aggregation, but it's very bad at local aggregation to speed up limited
connections like trying to bond multiple 1gig ports between san/nas and a
server for example.

~~~
obo
Exactly. With Multipath TCP, a single TCP connection can benefit from multiple
interfaces, which is not the case with bonding techniques. See
<http://multipath-tcp.org/pmwiki.php?n=Main.50Gbps> for measurements showing
Multipath TCP aggregating 6 10 Gbps interfaces.

------
peterwwillis
Are there some more detailed docs I haven't found? I have a dozen questions
about how their protocol reacts to common problems.

It seems like the kernel's just treating it as a "dumb" tcp connection that
rx's and tx's on more than one interface. In this case you're taking the
problems existing tcp connections have and multiplying them by the number of
links. SCTP works around these problems and makes _both the connection and the
delivery of arbitrary messages_ more reliable, though seemingly at a heavy
cost to performance.

~~~
jlgaddis
There are several related RFCs[0], and the IETF's "Data Tracker" page for the
MPTCP[1] working group has a lot of great documents as well.

[0]: <https://www.google.com/search?q=multipath+tcp+rfc> [1]:
<http://datatracker.ietf.org/wg/mptcp/>

------
shawnreilly
Awesome Project! I'm curious how this relates to the Mobile IP Standard (RFC
5944). Is Multipath TCP intended to Augment Mobile IP? Replace it? Not related
at all? I would be very interested to hear from the Developers / Researches
how they envision Multipath TCP working with (or replacing) other Standards.

I would also be interested in hearing what needs to be done on the Client End
for Multipath TCP to be supported; For example, say I was to build a Linux
Server with the Multipath TCP enabled Kernel (to serve my App), would the
Clients (likely Mobile Smartphones) also need to run an O/S with a Multipath
TCP enabled Kernel? I'm assuming the answer is Yes, but I'm kind of hoping I
missed something and the answer is No. Because if this is the case, then both
Android and iOS would need to support Multipath TCP (which means this may not
become mainstream for quite some time).

Also, I noticed the referenced PDF said the Technology was already being
Licensed (with a Non-Disclosure Agreement). This seems to imply it will not be
Open Source. I checked the Git Repo, and don't see anything about this. Are
the Developers / Researchers related to Multipath TCP intending to Open Source
this?

In any event, Thank you for this release, something fun to play with!

~~~
mad_fab
1\. The server and the client must be MPTCP-"aware", so the anwser is yes (but
look, someone is talking about a proxy up there, it's a solution to avoid this
problem).

2\. I'm in the team working in the Linux Kernel implementation and yes this is
open source! <http://multipath-tcp.org/pmwiki.php?n=Users.DoItYourself>

~~~
shawnreilly
Thank you!!!

------
jacob019
What a wonderful improvement to our old networking design. Can certainly see
the benefits of redundancy, but the throughput advantage is huge. It will
allow massively high throughput between servers by aggregating commodity
hardware, scaling out instead of up. >1Gb server hardware is ridiculously
expensive. I've tried interface bonding and the performance sucks. Now I can
get a cheap 5Gb link using 5 dirt cheap NIC's. Awesome.

------
alexchamberlain
Isn't this solving the wrong problem? Shouldn't IP support packets with
multiple addresses?

~~~
Nrsolis
How do you propose to modify all of the ASICs in routers that are optimized to
do lookups on packet headers at the speeds they need to achieve?

Routers today are doing 32x100GbE in a single chassis. That's an investment in
hardware that isn't going to go away any time soon.

SCTP already solved this problem. Multipath TCP is an also-ran.

~~~
vy8vWJlco
That's the clearest case I've seen for not buying "smart" hardware (firewalls,
routers, etc) that know any more of the higher levels of the stack than
necessary.

I think the parent comment is valid though. If we wind up holding 2 or more
TCP connections to multiplex, with failover, we are basically creating an
overlay network that will have the same needs as the lower levels. Namely,
routing, link cost metrics, fragmentation, etc... This is effectively what IP
attempts to add on top of 802.3.

With that in mind, it seems like it would be more faithful to the point
(possible, if not practical) to use the Ethernet+IP stack directly on top of
two TCP connections (providing simulated physical links), creating a throw-
away network segment between two hosts that could re-use existing routing and
bonding logic. I don't think Linux could handle that many virtual interfaces
or taps, but that feels like the right place for it...

~~~
jlgaddis
The devices handling 32x100GbE connections are only able to do that _because
of_ the very specialized ASICs that have been custom developed.

I'm not aware of any device that is software-based that could come remotely
close to handling that about of traffic. Most current PC-level hardware has
trouble w/ 2x10GbE links (if they can even support that).

~~~
vy8vWJlco
I agree that specialization is often necessary, but I would prefer to think in
terms of add-ons to general purpose commodity architectures. For example, GPUs
are specialized for the task of 3D math, but their particular purpose is
decided by the CPU. Similarly, FPGAs can be dynamically programmed for a
special circuit. Neither is ideal, but in hindsight I would have liked to see
something similar develop for the task of routing - and the economic pressure
applied by not buying ASICs (that break the stack seperation) would have
helped (yes we have benefited from them, but we have benefited from taking a
specific path - we don't know if there was a better one). Routing probably
would have benefited from an earlier and stronger push towards multicore
processors - only there was no reason to do it in consumer CPUs because we
view routing and computing largely as exclusive domains (even though the
Internet's success has come from connecting many general purpose endpoints...
Why is the fabric not also general purpose?).

Going forward, I suspect GPUs could also be applied to make routing decisions
with acceptable speeds. Have a look at [http://www.date-
conference.com/proceedings/PAPERS/2010/DATE1...](http://www.date-
conference.com/proceedings/PAPERS/2010/DATE10/PDFFILES/02.6_4.PDF) for some
trials.

~~~
mprovost
The specialised chips that are used by routers are CAMs and TCAMs. Basically,
associative arrays (hashes or dictionaries as they're called in most scripting
languages) implemented in hardware. You can program an FPGA to run as a CAM.
Mostly what they're doing are repeated lookups of where to switch or route
packets (based on header fields). If you're in a CPU world with RAM you can
try and optimise the data structure that holds the lookup tables so you get
decent speed but with CAM it's implemented in hardware so the lookup times are
fixed and known. It's not really a limit of computation that can be helped
with multiple cores, the CPUs can be pretty slow. Multithreading or parallel
processing is typically handled on routers by distributing the workload to
packet processors that handle a single port or a group of ports, all looking
at the same CAM (or copies of it in the more distributed chassis-based
architectures). Modern routers are basically supercomputers in a box. Of
course there's an inherent limit at the PHY level where you can only receive
or transmit a single packet at a time on an interface so you can't
multiprocess beyond a CPU per port basically which limits the complexity of
the design. 1000 cores on a GPU won't help you if you only have 100 ports to
route traffic on.

On a typical Intel PC, the MMU's TLB is implemented in CAM as well but it's
tiny in comparison to the ones in routers. CAMs are pretty power hungry
compared to RAM. They are becoming commodities though and the router
manufacturers are getting away from doing custom ASIC designs. Intel bought
Fulcrum which makes these packet processors and companies like Big Switch are
doing cool stuff with white box network hardware. It's really going to bring
the cost of the hardware down pretty dramatically in the next couple years.

~~~
vy8vWJlco
I'm looking forward to that!

------
mad_fab
If you want to try it; it's easy [http://multipath-
tcp.org/pmwiki.php?n=Users.HowToInstallMPTC...](http://multipath-
tcp.org/pmwiki.php?n=Users.HowToInstallMPTCP)

Also, we have a measurements campaign running, if you want to contribute ;)
<http://multipath-tcp.org/pmwiki.php?n=Users.AboutMeasures>

------
Zash
Whatever happened with SCTP?

~~~
koenigdavidmj
The only thing I've ever seen in it was a modification of FTP to use SCTP
streams on a single connection rather than have two connections. This would of
course make the whole active-FTP-over-NAT business easier, except for the fact
that the average NAT device ($50 home router) probably doesn't play nicely
with SCTP.

Given the subject matter, I should probably include my usual disclaimer
string: I work as a software engineer for F5 Networks. Anything I post here is
my own thought and is not intended to represent F5.

~~~
noselasd
When you use a mobile phone, chances are big most of the control plane traffic
(call setup, SMS, data channel setup/teardown, roaming signalling, phone
registring and so on) travel through an SCTP association somewhere.

------
yoran
What I especially like about this is that the project was developed in an
academic context while being hugely beneficial to society. I mean, improving
the performance of a fundamental protocol like TCP has huge effects. If only
we had more academic research like this one! Btw, also cool that it was
developed 15 minutes from my house.

