
Is the Network the Limit? Dealing with Weave, CoreOS and Azure - SanderMak
https://arjanschaaf.github.io/is-the-network-the-limit/
======
jbeda
I suggest you also look at GCE for network performance. It is something that
the Google networking team has been working hard on and, because of the unique
network fabric at Google, should scale to lots of VMs.

Also, you can get equivalent "IP per container" using advanced routing with
GCE. Flannel has a GCE specific back end where it inserts routes for each
subnet assigned to each range. That will eliminate the extra level of encap.

(Disclaimer: I started GCE but am no longer at Google.)

~~~
__monadic
+1 to @jbeda, but let me offer some comments for the avoidance of possible
confusion among potential Weave users.

Weave aims to deliver a completely _portable_ network. In other words if you
create an application using Docker container and a Weave network, that should
be able to run anywhere. And all this should be 'magically simple'. Once you
have decided where to run your app, you may wish to trade portability for
performance gains. For example by using GCE networking (or Azure, or ...).

To date Weave achieved portability by sending some packets (inter host)
through user space. This has big benefits in terms of ease of use eg. dealing
with wide area networks, multicast and firewalls. But under load it performs
worse than kernel-only models.

We now have Weave "fast data path" \-
[http://blog.weave.works/2015/06/12/weave-fast-
datapath/](http://blog.weave.works/2015/06/12/weave-fast-datapath/) ... This
aims to deliver close to line performance with portability and extreme ease of
use.

There may of course be yet more optimisations that end users wish to
investigate. If you are willing to sacrifice some portability you could
certainly make use of fast networks provided by a specific public cloud. We
haven't yet seen a strong need to support this, but it is certainly a
reasonable thing.

(Disclaimer: I work at Weaveworks)

------
jamescun
Is the poor performance of Weave really surprising? Userland networking will
always be lesser performing than that of native kernel networking (the show is
somewhat better with technologies such netmap/PF_RING/DPDK). Weave itself
would probably improve in performance if it utilised a TUN/TAP device + static
routes as opposed to its current mangling of packets with pcap and BPF.

~~~
dpw
We did try TUN/TAP. It seemed to be slower than pcap.

Our answer to weave performance concerns is on its way:
[http://blog.weave.works/2015/06/12/weave-fast-
datapath/](http://blog.weave.works/2015/06/12/weave-fast-datapath/)

We're really interested to hear how weave fast datapath works out in all kinds
of environments, which is why we put the preview out. It's a shame that
Arjan's post did not include numbers for FDP, but maybe he'll be able to
include them in an update.

~~~
justincormack
Indeed tun/tap performance is terrible, it seems to take about a whole CPU at
gigabit.

~~~
jsolson
That, in general, is not the behavior I've observed on native hardware.
Without getting into hard numbers, I routinely see multiple
gigabits/second/core.

~~~
justincormack
With which options? Standard tap (or tun?), or macvtap? The last test I did
(macvtap to an external interface but none of the offload options), a gigabit
used a lot of CPU, like around a whole CPU just copying data.

